One of the simplest pieces of test-verification: Add a test task to run desktop_unittest.py with '--verify' as an argument. Bug 1380122 will add support for --verify. For now, I'm making this a tier 3 job. I want to observe it for at least a few days before upgrading to tier 2 (or maybe 1!). For now, I'm making this linux-only. I think windows and mac support will follow soon after. Android might be included one day. I won't check this in until bug 1380122 is resolved. https://treeherder.mozilla.org/#/jobs?repo=try&revision=911d9abe6ed23dc7e97ce1784b26ca13aba7f605&filter-tier=1&filter-tier=2&filter-tier=3

Comment on attachment 8893604 [details] [diff] [review] add test-verify, tc(TV) task Review of attachment 8893604 [details] [diff] [review]: ----------------------------------------------------------------- a few small things right now ::: taskcluster/ci/test/test-sets.yml @@ +211,4 @@ > - mochitest-webgl > - reftest > - reftest-no-accel > + - test-verify I think linux32 might not be that useful ::: taskcluster/ci/test/tests.yml @@ +1692,5 @@ > + description: "Extra verification of tests modified on this push" > + suite: test-verification > + treeherder-symbol: tc(TV) > + loopback-video: true > + e10s: both we only run in e10s mode now, so remove the e10s line and it will default to true. @@ +1711,5 @@ > + linux.*: > + - unittests/linux_unittest.py > + - remove_executables.py > + extra-options: > + - --verify how do we specify this is mochitests vs reftest/xpcshell/wpt ?

Thanks for your initial comments. To provide more context: Bug 1380122 - mostly working, but needs some cleanup and attention to some auxillary features - will add "verify mode" to mozharness. In verify mode, the mozharness script will download test packages for all supported test types, examine the changesets for the push to identify changed files, identify which changed files are test files (appear in manifests for supported test types), and then run the applicable test harness for each test file, in verify mode. Note that one test-verify task will run one mozharness script, which may run multiple test harnesses (mochitest, xpcshell, reftest, etc) and or test configurations (mochitest-plain vs mochitest-bc, etc). In this version of the patch, I have removed linux32. Test verification works fine there, but you are right, there's limited incremental value there, over linux64. I removed the e10s specification, so we'll default to e10s only. I was going to run in both e10s and non-e10s to allow for xpcshell, and perhaps other non-e10s-only tests. That is a little wasteful though, especially in an all-e10s browser testing world, so I'm going to look at supporting non-e10s harnesses in an e10s mozharness environment - I expect that will be worked out in bug 1380122. I also added a 'files-changed' filter: when no js/html files have been modified, we can save some resources by skipping the verify task.

Comment on attachment 8893917 [details] [diff] [review] add test-verify, tc(TV) task Review of attachment 8893917 [details] [diff] [review]: ----------------------------------------------------------------- assuming the when clause works properly :)

I've added https://developer.mozilla.org/en-US/docs/Test_Verification, linked to https://developer.mozilla.org/en-US/docs/Mozilla/QA/Automated_testing, in partial fulfillment of Tier 1/2 requirements.

:kwierso -- I'm ready to check this in now, adding a tc(TV) job to linux64-opt/debug/asan. The new job is tier 3 for now. I have docs started (comment 5) and I take responsibility for the new job (jmaher to backup). Just checking, do you have any concerns?

No concerns for it to be tier-3.

Pushed by gbrown@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/e7385dfdb85a Add a test task for modified test verification; r=jmaher